Repository navigation
feat(cloudfetch): honor download settings with the kernel - #566
aakash-saravanan-db wants to merge 6 commits into
Conversation
There was a problem hiding this comment.
Verdict: 1 Low
Looks good: one low test-coverage gap. The kernel's forwarding precedence (positive explicit option > case-insensitive raw session parameter > MaxDownloadThreads when set) is implemented correctly in buildKernelConfig, DeepCopy carries the new flag, and values are normalized before forwarding. Possible follow-up, not caused by this PR: when both an uppercase raw key and WithKernelMaxChunksInMemory are set, both keys still reach the kernel.
| query string | ||
| set bool | ||
| }{ | ||
| {"", false}, |
There was a problem hiding this comment.
🔵 Low — TestMaxDownloadThreadsDSNPresence only has cases where set is false ("" and ?maxDownloadThreads=), so the if tc.set { assert.Equal(t, "10", value) } branch never runs. That leaves the path this PR adds untested: DSN maxDownloadThreads=N → MaxDownloadThreadsSet → cloudfetch_max_chunks_in_memory=N in the kernel SessionConf. The DSN subtest in TestMaxDownloadThreadsNonPositiveDegradesToDefault asserts the flag but never calls buildKernelConfig. Add a set case such as {"?maxDownloadThreads=4", true} and assert the forwarded value (the hard-coded "10" would then also need to follow the case's value), or a ?maxDownloadThreads=0 case that expects "10".
There was a problem hiding this comment.
Verdict: 1 Low
Looks good. There is one low finding: the CHANGELOG entry doesn't flag that existing maxDownloadThreads settings now change kernel memory and throughput. The forwarding logic matches what the docs describe and the tests cover it: an explicit value is forwarded, the default of 16 stays when unset, a raw session parameter in any letter case takes precedence, WithKernelMaxChunksInMemory takes precedence over both, and the caller's map is not changed.
There was a problem hiding this comment.
Verdict: 1 Medium
Looks mostly good, with 1 medium concern: when a positive WithKernelMaxChunksInMemory is combined with a raw chunk session parameter cased differently (e.g. uppercase), both keys reach the kernel, so the documented precedence isn't enforced. Forwarding maxDownloadThreads via the new MaxDownloadThreadsSet flag, the DSN/option presence tracking and DeepCopy handling all look correct and are covered by tests.
There was a problem hiding this comment.
Verdict: 1 Low
Mostly good, with one low concern: the documented case-insensitive precedence isn't enforced when an explicit override and a mixed-case raw key are both set. Forwarding maxDownloadThreads only when it is explicitly set, plus the DSN presence tracking, looks right and is well tested. Nit: the telemetry MaxChunksInMemory comment was reworded, but the field is still never populated.
There was a problem hiding this comment.
Verdict: 2 Nit
Looks good: no correctness concerns found. I traced how the setting flows through the code. MaxDownloadThreadsSet is set by both the option and ParseDSN; an empty DSN value leaves it unset because extractAsInt only reports values that are present and non-empty. withUserConfig and DeepCopy carry it through. Non-positive values are changed to 10 in NewConnector before buildKernelConfig runs. The precedence is checked against session-parameter keys that differ only in case, and the explicit kernel option is still applied last. One trade-off is already documented in the CHANGELOG: an explicit maxDownloadThreads=10 (common in sample DSNs) now lowers the kernel window from 16 to 10. The same applies to Reyden auto-fallback connections.
Other findings
- ⚪ Nit —
telemetry/request.go:MaxChunksInMemoryis never filled in by the driver (the kernel owns telemetry), so the new comment "zero means the value was not reported" is accurate. It could also say the effective window may now come frommaxDownloadThreads. - ⚪ Nit —
kernel_config_test.go: the rewritten "unset download window" subtest only coversKernelExperimentalset to a non-positive value. The old case with nilKernelExperimentalis now covered only indirectly, byTestMaxDownloadThreadsDSNPresence. Adding the nil case back to the loop would keep the guard local.
| func WithMaxDownloadThreads(numThreads int) ConnOption { | ||
| return func(c *config.Config) { | ||
| c.MaxDownloadThreads = numThreads | ||
| c.MaxDownloadThreadsSet = true |
There was a problem hiding this comment.
why do we need this in Go but not Python and Node?
There was a problem hiding this comment.
Go initializes MaxDownloadThreads to 10 even when omitted, so this flag distinguishes “unset” from “explicitly set to 10.”
Python can detect omission through its keyword arguments. Node tracks explicit assignments through its config property accessor.
All three now preserve the kernel default of 16 when unset and honor explicit values, including 10. The flag is Go’s way of preserving that distinction.
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
ac2504c to
03c875f
Compare
Feature or gap
Go applications could set
WithMaxDownloadThreadsormaxDownloadThreadsin the DSN, but kernel connections ignored the setting.What this PR addresses
Forward explicit standard settings into the existing kernel download window; keep the kernel default of 16 when unset. Non-positive explicit values normalize to 10. Preserve explicit kernel/raw overrides and document the 256 ceiling, connection timing and additional consumer buffers. A positive
WithKernelMaxChunksInMemorywrites the canonical lowercase key; the kernel applies case variants in sorted order, so it takes precedence over raw spellings and strips all of them before server requests.Dependencies
Existing native kernel APIs; no native API or version bump is required for this mapping.
Tests and validation
CGO_ENABLED=0 go test ./...passes. Built-driver checks verify explicit settings, DSN parsing, precedence, the unset default, complete results and reuse. After equal warmups and five alternating runs per side, using a native built from kernela488a05, DBSQL submission-through-drain medians for 3M ordered rows: Thrift 6.33 s → kernel 4.15 s (-34.4%).